Summary

Improves async generation throughput, model-slot utilization, and large-run scheduler scaling with four targeted changes. The implementation is net +16 nonblank production LoC and adds no public API, dependency, feedback controller, or user tuning knob.

Changes

• Make bounded-borrow admission work-conserving: a sole runnable scheduling group can use all resource slots, while existing borrow debt gives a newly waiting peer priority as slots are released.
• Bound fair-queue priority state by active scheduling groups instead of retaining historical task entries.
• Compact checkpointed completion state after a row group is durably written and all of its workers finish.
• Write resume metadata after the first durable row group and refresh it at completion; parquet files remain the source of truth for resume progress.

Performance vs. main

End-to-end throughput and utilization

Full-job reservation measurements count each physical endpoint once, including shared-model endpoints. The reported unit is endpoint concurrent-service-capacity slot-hours, a GPU-hours proxy until the inference owner supplies the endpoint-to-GPU allocation.

Work-conserving admission removes speculative idle capacity while remaining practically equivalent for both slow-before-fast and fast-before-slow calibrated flows. It cannot preempt work already running: in the measured worst case, a late peer started up to 350 ms later when it arrived behind a 400 ms task; borrow debt still gives it the next released slot.

Large-run scheduler and memory scaling

The pathological one-row-group-per-record case retains one compact terminal marker per group: 70.511 MiB at one million groups. With normal buffer sizes, retained completion state is effectively constant relative to record count.

Metadata checkpointing

The shared-endpoint and hourglass cells are practically equivalent on wall time within a ±2% margin, while the heterogeneous fork/join cell improves both throughput and utilization.